A Nonlinear Analytic Center Cutting Plane Method for a Class of Convex Programming Problems

A cutting plane algorithm for minimizing a convex function subject to constraints de ned by a separation oracle is presented. The algorithm is based on approximate analytic centers. The nonlinearity of the objective function is taken into account, yet the feasible region is approximated by a containing polytope. This containing polytope is regularly updated either by adding a new cut at or by shifting an existing cut to the test point. In the rst phase of the algorithm, the test point is an approximate analytic center of a containing polytope. In the second phase, it becomes an approximate analytic center of the intersection of a containing polytope and a level set of the nonlinear objective function. We prove that the algorithm converges and establish its complexity. In the case where the oracle generates a nite number of cuts, the algorithm is polynomial in this number, while it is a polynomial approximation scheme (in the dimension of the space) in the case where the oracle can generate an in nite number of cuts. Key words. Convex programming, interior-point methods, analytic center, cutting planes, potential function, self-concordance. R esum e Cet article d ecrit un algorithme de plans coupants bas e sur le concept de centre analytique approch e, appliqu e au probl eme de la minimisation d'une fonction convexe satisfaisant des contraintes donnees par un oracle de s eparation. La nonlin earit e de l'objectif est prise en compte explicitement dans la d e nition du centre analytique, tandis que la r egion admissible est approxime e par un polytope circonscrit. La description de l'ensemble admissible approch e par un polytope circonscrit est ra n ee par l'addition de coupes qui passent par le point courant ou par la translation au point courant d'une coupe existante. Durant la phase I, le point courant est de ni comme le centre anlytique approch e du polytope circonscrit. Durant la phase II, ce point est le centre analytique de ni par l'intersection du polytope circonscrit et une corbe de niveau de l'objectif non lin eaire. Nous d emontrons la convergence et la complexit e de l'algorithme. Dans la cas o u l'oracle g en ere un nombre ni de coupes, l'algorithme est polynomial en ce nombre, tandis que l'algorithme devient un sch ema d'approximations polynomial (en la dimension de l'espace) si l'oracle a potentiellement un nombre in ni de coupes. 1 Introduction Recently, many cutting plane or column generation algorithms based on analytic centers have been studied. Sonnevend [22] rst introduced the concept of analytic center of a system of convex inequalities. Go n, Haurie and Vial [11] and Ye [25] proposed a cutting plane algorithm that uses the approximate analytic center of the system of inequalities generated in the previous iterations to add a new cut. This method performs well in practice [3, 4, 10, 11, 12]. Later, Go n, Luo and Ye [13, 14] analyzed the complexity of this analytic center cutting plane method for solving a convex feasibility problem de ned by a separation oracle. Go n and Vial [15] have extended this result by considering shallow, deep and very deep cuts. Complexity bounds have also been established for several related methods, by Nesterov [20], Atkinson and Vaidya [2], Kiwiel [16] and Altman and Kiwiel [1]. Extensions to the case of multiple cuts are given by Ye [26]. Luo and Sun [17] have used an analytic center based column generation algorithm for convex quadratic feasibility problem that uses quadratic cuts. They have also extended the case to self-concordant convex inequalities using convex cuts [18]. Interior-point and cutting plane methods have also been combined in a di erent way by Den Hertog et al. [6]; they proposed a logarithmic barrier cutting plane method for convex (possibly nonsmooth, semi-in nite) programming and provided some computational results. Further computational results for the same method were later given by Kaliski et al. [7]. Although this method is of interior point nature, it is not an analytic center cutting plane method. In this paper, we consider the problem (CP )( min f(y) y 2 where f : [0; 1]m ! R is convex and C3-smooth and Rm is a convex set with a nonempty interior contained in the cube 0 = [0; 1]m. The set is de ned implicitly by a separation oracle which for every y 2 0 either answers that y belongs to interior of or generates a separating hyperplane fy 2 Rm : aTy aT yg . Without loss of generality, we assume that a is normalized so that kak = 1. The set can be de ned by, e.g., nite or in nite number of pseudoconvex constraints, nite or in nite system of convex inequalities where some or all the constraints are nonsmooth, or by an in nite or large number of linear inequalities. We also assume that for any z (an upper bound on the objective function) 1 'z(y) = ln(z f(y)) is -self-concordant on (0; 1)m. That is for all h 2 Rm and for all y 2 (0; 1)m, fD3'z(y)[h; h; h]g2 4 (hTr2'z(y)h)3; where D3'z(y)[h; h; h] denotes the third di erential of 'z at y and h. We also assume without loss of generality that 1. The assumption that is independent of z is not restrictive, for we can conclude, using Proposition 2.1.1 (ii) of Nesterov and Nemirovskii [21], that if (y; z) = ln(z f(y)) is -self-concordant on f(z; y) : y 2 (0; 1)m; z > f(y)g, then 'z is -self-concordant on (0; 1)m for any z > f(y). Problems formulated as (CP ) appear a lot in practice. For example, many multicommodity ow problems give rise to very large nonlinear programs with linear constraints [10]. We consider an analytic center based column generation algorithm for solving problem (CP ) that takes the nonlinearity of the objective function into account. The algorithm consists of two phases. The rst phase starts with 0 = [0; 1]m as the original outer approximation of the feasible region. Then using a central cutting plane method, an initial feasible point is found from which we nd an initial upper bound on the optimal value, say z(0). Then the nonlinear cut fy 2 Rm : f(y) z(0)g corresponding to the objective is added. This de nes a \set of localization" that contains the solution set to the problem (CP ). The second phase of the algorithm consists of inner and outer iterations. At each inner iteration, an approximate analytic center of the set of localization is calculated and tested by the oracle for feasibility. If the test point is not feasible either a linear cut will be added that goes through the test point or if a cut parallel to the current cut has already been added, the older cut will be translated all the way to the current test point. The inner iterations continue until a feasible test point is found. If the feasible test point is an approximate solution of (CP ), the algorithm stops. Otherwise, an outer iteration is performed in which the value of the upper bound on the objective is reduced. As the number of iterations increases, the set of localization shrinks and the algorithm eventually nds an approximate solution of the problem. Our algorithm can be thought of as an extension of the central cutting plane algorithm of Go n, Luo and Ye [14] for the convex feasibility problem. However, we consider the problem in the form of an optimization problem involving a nonlinear objective. The important feature of this extension is that a nonlinear cut corresponding 2 to a level surface of the objective is considered in our case, whereas only linear cuts are used in their algorithm. Our algorithm also allows translation of linear feasibility cuts all the way to the approximate analytic center. The central cutting surface algorithm of Luo and Sun [18], on the other hand, uses convex cuts, yet the feasible set that they consider is de ned by a nite number of convex di erentiable inequalities that admits a self-concordant logarithmic barrier. In our case, however, the feasible region is implicitly de ned by a separation oracle, and in this sense our case considers a broader class of problems. This paper is organized as follows. In Section 2, we review some basic concepts and results to be used in the subsequent analysis. In Section 3, we give our column generation algorithm. In Section 4, some results needed for complexity of the algorithm will be given. Finally, complexity and convergence of the algorithm will be discussed in Section 5 and Section 6. We will adopt the standard notations used in the interior point literature. For any generic vector u, uT will denote the vector transpose, and the corresponding capital letter U will denote the diagonal matrix whose i-th diagonal element is given by the i-th component of u. The vector ~e will represent the vector of ones of appropriate dimension. For a real number a, dae represents the smallest integer greater than or equal to a. The inradius of a compact convex set G in Rm denoted by (G) is the radius of the largest sphere contained in G. The interior of G will be represented by Gint. For two symmetric m m-matrices A and B we write A B if the matrix A B is positive semide nite. We will also use the following notation: kukH = hu;Hui1=2; kuk H = hu;H 1ui1=2; whereH is a given symmetricpositive de nite matrix and h ; i represents the standard inner product in Rm. Throughout this paper, we assume that f is the optimal value of (CP ). 3 2 Preliminaries Let be a bounded set in Rm de ned by n convex inequalities as follows: = fy 2 Rm : hj(y) 0; j = 1; : : : ; ng; (1) where hj(y); j = 1; : : : ; n, have continuous rst and second derivatives. Suppose that int is nonempty. The potential function of is de ned as ( ; y) = n X j=1 ln[ hj(y)] (2) where hj(y) is the slack of the j-th inequality. We recall (see [21, De nition 2.1.1]) that a function is strongly -self-concordant on int if it is -self-concordant on int and, in addition, tends to in nity along every sequence fyi 2 intg converging to a boundary point of int. Hence by its de nition, ( ; y) will be automatically strongly -self-concordant whenever it is -self-concordant. The gradient and the Hessian of ( ; y) are given by g( ; y) = r ( ; y) = n Xj=1 rhj(y) hj(y) and H( ; y) = r2 ( ; y) = n Xj=1(rhj(y)rhj(y)T hj(y)2 + r2hj(y) hj(y) ) : The Newton direction of ( ; y) at y will be given by q( ; y) = H( ; y) 1g( ; y): The min-potential of is de ned as P ( ) = min y2 int ( ; y): (3) The unique point in int where this minimum is attained is called the \analytic center" of . Let ya be this analytic center. We recall from [21] that one proximity measure between the analytic center ya and a given y 2 int is p kg( ; y)k H( ;y): 4 Note that kq( ; y)kH( ;y) = kg( ; y)k H( ;y) : Another measure is p ky yakH( ;y): Finally, the \gap" ( ; y) P ( ) is a third measure of proximity. De nition 2.1 Let 0 < < 1. We say y 2 int is a -approximate analytic center of if p kg( ; y)k H( ;y) : The next two lemmas relate the given proximity measures. Lemma 2.1 Let y 2 int and ( ; y) be -self-concordant on int. Let 2 [0; (p2 1)2]. If y is a -approximate analytic center of , then p ky yakH( ;y) (2 p2) ; where ya is the exact analytic center of . Proof. See Nesterov [20, Corollary 2.1]. Lemma 2.2 Let y 2 int and ( ; y) be -self-concordant on int. If y is a (1=3)approximate analytic center of , then ( ; y) P ( ) (kg( ; y)k H( ;y))2 1 f94p kg( ; y)k H( ;y)g2 : Proof. See Den Hertog [5, Lemma 3.14]. 2 The following lemma is due to Nesterov and Nemirovskii [21, Proposition 2.2.2] and states that a su cient decrease in the potential function can be obtained by taking a step of suitable size along the Newton direction. 5 Lemma 2.3 Suppose that y 2 int and ( ; y) is -self-concordant on int. Let = 1=f1 +p kg( ; y)k H( ;y)g: Then y+ = y + q( ; y) 2 int and ( ; y) ( ; y+) fp kg( ; y)k H( ;y) ln(1 +p kg( ; y)k H( ;y))g= : The following result is part of Theorem 2.2.3 of [21]. Lemma 2.4 Suppose that y 2 int and ( ; y) is -self-concordant on int. Let = 2 p3 : (4) If p kg( ; y)k H( ;y) < , then y+ = y + q( ; y) 2 int and p kg( ; y+)k H( ;y+) 8<: p kg( ; y)k H( ;y) 1 p kg( ; y)k H( ;y)9=;2 : (5) Remark. Lemma 2.4 implies that if y 2 int satis es 0 < < p kg( ; y)k H( ;y) < ; then we can take full Newton steps to calculate a -approximate analytic center of . We note that, the quantitiesp kg( ; yi)k H( ;yi), i = 0; 1; : : : ; with y0 = y, by (5), decrease quadratically. Therefore, the total number of Newton iterations to nd a -approximate analytic center of , starting at y, is at most N = & 1 ln 2 ln ln 2 ln(1 ) ln 2 ln(1 )!' ; (6) an O(1) number depending on only. De nition 2.2 A mapping Tu from the convex subset U of a linear space onto the class of subsets of Rm is called a concave array if u1; u2 2 U; 2 [0; 1] implies T u1+(1 )u2 Tu1 + (1 )Tu2 : 6 Let = fy 2 : f(y) f + g; (7) where 0. Lemma 2.5 The mapping is a concave array on R+. Proof. The proof is elementary and follows from the convexity of f , and the de nition of a concave array. 2 For more information on concave arrays and behaviour of level sets of a convex function, we refer the reader to Eggleston [8] and Go n [9]. Theorem 2.1 Let 0 1. Then ( ) ( ): Proof. By Lemma 2.5, we have + (1 ) 0: Let y be any optimal solution of (CP ). Then y ( y ) + (1 )( 0 y ): Now 0 2 (1 )( 0 y ). Hence ( y ) + (1 )( 0 y ) ( y ): This implies that y ( y ) or ( y ) ( ( y )) ( y ): Using the trivial equality (G) = (G t); which holds for any compact convex set G 2 Rm and any vector t 2 Rm, we obtain ( ) ( ): 7 Corollary 2.1 Let w0 2 , z(0) = f(w0) and > 0 be a given tolerance. Suppose z(0) f . Then ( ) ( z(0) f ) z(0) f : Proof. In Theorem 2.1, let = z(0) f and = : Corollary 2.2 Let 0 < 1. Then ( ) ( 1): Proof. In Theorem 2.1, let = 1 and = : 2 We will make the following assumptions throughout this paper. Assumption 1: The interior of contains a full-dimensional closed ball of radius 0 < < 1=2. Assumption 2: The interior of 1 contains a full-dimensional closed ball of radius 0 < < 1=2. Remarks. Assumption 1 implies that the feasible region in not too narrow, and Assumption 2 implies that 1 is not too narrow. As we will see later Assumption 1 guarantees that in O ((m2= 2) ln(1= )) (where the asterisk notation means the lower order terms are ignored), a feasible solution of is found. Assumption 2, in view of Corollary 2.2, will be used in the rst stopping rule of the algorithm to guarantee that the total number of feasibility cuts produced by the algorithm is bounded and is in fact O (m2=( )2). Here > 0 is the accuracy of any approximate optimal solution at termination, i.e., if y is any such approximate solution, then y 2 . We will refer to any y 2 as an -solution of (CP ). 3 The Column Generation Algorithm In this section, we rst explain some notation that will be used in the algorithm and later in its complexity analysis. In phase I of the algorithm, the set j;t represents the outer approximation of when the j-th cut is added and after which t translations 8 of existing cuts are performed, and (j) represents the maximum number of these translations after addition of the j-th cut. The approximate analytic center of j;t is represented by yj;t. The (j + 1)-th cut added in phase I is represented by fy : (~ aj+1)Ty ~ cj+1(j + 1; 0)g where ~ cj+1(j + 1; 0) = (~ aj+1)Tyj; (j). Furthermore, in j;t, the constraint fy : (~ ai)Ty ~ ci(j; t)g represents a cut added during the i-th iteration (i j) and translated zero or several times so that its constant during the (j; t)-th iteration of phase I is ~ ci(j; t). We refer to the total number of feasibility cuts produced in phase I, k(0; 0), as J (for simplicity) as well. After a feasible point is found, the objective constraint, i.e., fy : f(y) z(0)g, is added to this nal outer approximation in phase I, J; (J), to get 0;0 0 . In phase II, translations of the nonlinear objective constraint will be performed as well. By the objective constraint, we mean the constraint represented by the level set fy : f(y) z(l)g in the l-th outer iteration. In this phase, j;t l represents the \set of localization" when the objective constraint is translated for the l-th time and for that l, the j-th cut is added and after the addition of this j-th cut, t translations of existing feasibility cuts are performed. The approximate analytic center of j;t l is represented by yj;t l . The maximum number of cuts added during the l-th objective translation is represented by (l) and the maximumnumber of constraint translations performed during the (l; j)-th iteration is represented by T (l; j). The (j + 1)-th cut added in the l-th outer iteration of phase II is represented by fy : (aj+1 l )Ty cj+1 l (l; j + 1; 0)g where cj+1 l (l; j + 1; 0) = (aj+1 l )Tyj;T (l;j) l . In addition, in j;t l , the constraint fy : (avu)Ty cvu(l; j; t)g represents a cut added during the (u; v)-th iteration (before the (l; j)-th iteration) and translated zero or several times so that its constant during the (l; j; t)-th iteration is cvu(l; j; t). The constraint fy : (~ ai)Ty ~ ci(l; j; t)g, on the other hand, is a cut added during the i-th iteration (i J) in phase I and translated zero or several times so that its constant during the (l; j; t)-th iteration of phase II is ~ ci(l; j; t). For simplicity, the set (l);T (l; (l)) l is represented by fl , and the feasible point found in the l-th objective translation which is the approximate analytic center of fl is represented by yf l . Finally, we assume the 2m hyperplanes of the box 0 are never translated; they are remained untouched. We assume so for two reasons. The rst reason is that considering the possible translations of these hyperplanes causes only minor changes in the proofs and does not change the complexity bounds. The second reason is to avoid lengthening the proofs and introducing extra notation. We now describe the algorithm. 9 Phase I: (Finding an initial interior feasible point) Step (1.0) Set 0 = [0; 1]m, 0 < < 1, 0 < < 1=2, 0 < < 1, 0 < (p2 1)2=2, y0 = (1=2)~e, c( ; ) = 1 (p2 lnp2 1)= + ln[((p2 1)2 )=p ]: Step (1.1) Query the oracle to see whether y0 2 int. If y0 2 int Then set w0 = y0; goto Step (1.4). Else add the cut returned by the oracle, i.e., update 1;0 = 0 [ f(~ a1)Ty (~ a1)T~e=2g. Set j = 1 and t = 0. Endif Step (1.2) Find a -approximate analytic center of j;t, yj;t. Step (1.3) Query the oracle to nd out whether yj;t 2 int. If yj;t 2 int Then set w0 = yj;t. Else the oracle returns the vector a. -Case 1: (A parallel cut has been added before.) If there is 1 |̂ j such that a = ~ a|̂, update ~ c|̂(j; t+ 1) = (~ a|̂)Tyj;t and keep the constants of other cuts the same to get j;t+1. Set t := t+ 1 and go to Step (1.2). -Case 2: (No parallel cut has been added before.) Set ~aj+1 = a; add the cut fy : (~ aj+1)Ty ~ cj+1(j + 1; 0)g and keep the constants of the previous cuts the same to get j+1;0. Set (j) = t; j := j + 1; t = 0 and go to Step (1.2). Endif Step (1.4) Set ~ r = hrf(w0); fH( j; (j); w0)g 1rf(w0)i1=2; ( ; ) = ln ~ r ln p + 2 p (2 p2) ; z(0) = f(w0) +p ~ r= ; 0;0 0 = j; (j) [ fy : f(y) z(0)g; k(0; 0) = j; l = 0; j = 0; t = 0; yf 1 = w0. End (Phase I) 10 Phase II: (Finding an -solution of (CP )) Step (2.1) Find a -approximate analytic center of j;t l , yj;t l . Step (2.2) Check (by querying the oracle) if yj;t l 2 int. If yj;t l 2 int Then set (l) = j; T (l; j) = t; yf l = yj;t l . Else the oracle returns the vector _ a. -Case 1: (A parallel cut has been added before.) If there is (l̂ ; |̂) from the previous iterations such that _ a = a|̂̂l or there exists ~ai from phase I such that _ a = ~ai, update c|̂̂l(l; j; t+ 1) = (a|̂̂l)Tyj;t l or update ~ ci(l; j; t+ 1) = (~ ai)Tyj;t l and keep the constants in the other linear cuts the same to get j;t+1 l . Set t := t+ 1 and return to Step (2.1). -Case 2: (No parallel cut has been added before.) Set aj+1 l = _ a, add the cut f(aj+1 l )Ty cj+1 l (l; j + 1; 0)g, and keep the constants in the previous linear cuts the same to get j+1;0 l . Set T (l; j) = t; k(l; j + 1) := k(l; j) + 1; j := j + 1; t = 0. If (stopping criterion 1) 1+4m ln(1+ k(l;j) 8m2 ) 1+2m+k(l;j) exp 2c( ; )k(l;j) 1+2m+k(l;j) 2 ( ; ) 1+2m+k(l;j) ( )2 2m Then stop (yf l 1 is an -solution). Else return to Step (2.1). Endif Endif Step (2.3)If (stopping criterion 2) z(l) f(yf l ) 4 5(1+2m+k(l; (l))) Then stop (yf l is an -solution). Else set z(l+ 1) = z(l) (z(l) f(yf l )) and keep the constants in the linear cuts the same to get 0;0 l+1. Set k(l+ 1; 0) = k(l; (l)); j = 0; t = 0. Set l := l + 1 and return to Step (2.1). Endif End (Phase II) 11 Remark. Note that since the feasibility cuts added or translated in the algorithm are all linear, the parameter of self-concordance of the potential function for any set of localization during phase II remains equal to (see, e.g., Nesterov and Nemirovskii [21, Proposition 2.1.1]). Note that this parameter is always equal to 1 in phase I. 4 Potential Increase We consider the changes in the min-potential in the following three cases: 1. When either the objective or an existing feasibility cut is translated; 2. when the nonlinear cut corresponding to the objective function is added; and 3. when a linear feasibility cut is added. In this section, we assume that the set is de ned by (1) and ( ; y) is -selfconcordant on int. The following lemma relates the slacks of the j-th inequality of evaluated at the exact and an approximate analytic center of . Lemma 4.1 Let ya and yc be the exact and -approximate analytic centers of respectively, where 2 [0; (p2 1)2]. Then 0 < hj(ya) 1 + (2 p2)p !hj(yc); j = 1; : : : ; n: (8) Proof. Let 1 j n be xed. Since kg( ; yc)k H( ;yc) =p , it follows from Lemma 2.1 that kya yckH( ;yc) (2 p2)p : But rhj(yc)rhj(yc)T hj(yc)2 H( ; yc): Therefore, jhrhj(yc); ya ycij hj(yc) (2 p2)p 12 or hj(ya) hj(yc) 1 jhrhj(yc); ya ycij hj(yc) (2 p2)p which implies (8). Corollary 4.1 Let be updated to as follows: = fy 2 Rm : hj(y) 0; j = 1; : : : ; n 1; hn(y) hn(yc)g where yc is a -approximate analytic center of with 2 [0; (p2 1)2] and 0 < 1. Suppose ( ; y) is -self-concordant on ( )int as well. Then P ( ) P ( ) + ( ; ) where ( ; ) = 1 + (2 p2)p 1. Proof. The proof of Luo and Sun [18, Lemma 3.1] also works here, using Lemma 4.1. 2 Remark. Note that Corollary 4.1 can be applied to both cases where either the objective or a linear cut is translated. The next lemma is a combination of Lemma 3.2 and Lemma 6.2 of Luo and Sun [18]. It shows the changes in the min-potential when a nonlinear cut is added and the proximity of the old approximate center to the new analytic center. Lemma 4.2 Let yc 2 int be a -approximate analytic center of with 0 < (p2 1)2=2. Suppose is updated to = \ fy 2 Rm : hn+1(y) hn+1(yc) +p r= g; where r = hrhn+1(yc);H( ; yc) 1rhn+1(yc)i1=2, so that ( ; y) is -self-concordant on ( )int as well. Then P ( ) P ( ) ln r ln p + 2 p (2 p2)! ; (9) and kg( ; yc)k H( ;yc) kg( ; yc)k H( ;yc) + p : (10) 13 Remark. Note that Lemma 4.2 applies to the last iteration of phase I in which the objective cut is added. Finally, we consider the changes in the min-potential when a feasibility cut is added. The following lemma is an extension of Lemma 2.1 of Nesterov [20]. There, the result is given for strongly 1-self-concordant functions, yet it can be extended to strongly -self-concordant functions (our ( ; y)) as well. The proof is given in the appendix. This lemma is also an extension of Ye [25, Theorem 2] which gives a lower bound on the min-potential for a reduced polyhedron. Lemma 4.3 Let yc be a -approximate analytic center of with 2 [0; (p2 1)2). Consider the new set + = fy : hi(y) 0; i = 1; : : : ; n; (a+)Ty (a+)Tycg with a+ 2 Rm such that ka+k = 1. Let y+ = yc [H( ; yc)] 1a+ 3p ka+k H( ;yc) : (11) Then y+ 2 ( +)int and ( ; yc) ln ka+k H( ;yc) + c( ; ) P ( +) ( +; y+) ( ; yc) ln ka+k H( ;yc) + c( ; ) (12) where c( ; ) = 1 (p2 lnp2 1)= + ln[((p2 1)2 )=p ]; (13) c( ; ) = =(3 ) + ln(3p ) (ln(2=3) + 1=3)= : (14) Remark. As a consequence of Lemma 4.3 and using the same notation and assumptions, we have the following changes in the min-potential while adding a linear feasibility cut. P ( +) P ( ) ln ka+k H( ;yc) + c( ; ): (15) 14 5 Convergence and Complexity Analysis In this section, we will analyze the convergence and complexity of the cutting plane algorithm. We give a detailed analysis for the overall complexity of the algorithm. The complexity analysis of phase I is similar and simpler and will be brie y discussed at the end of the section. The complexity of phase II will be the di erence of the two. We note that the feasibility cuts added in the algorithm are added to the sets 0 and j; (j) for j = 1; : : : ; J 1, in phase I, and to the sets v;T (u;v) u for u 0, v = 0; : : : ; (u) 1, in phase II. For simplicity, we will use the following notation in the rest of this section: yv u = yv;T (u;v) u ; Hv u = r2 ( v;T (u;v) u ; yv u); rv u = hav+1 u ; (Hv u) 1av+1 u i1=2; Hj = r2 ( j; (j); yj; (j)); rj = h~ aj+1; (Hj) 1~ aj+1i1=2; where u 0, v = 0; : : : ; (u) 1 and j = 0; : : : ; J 1. We recall from Section 3 that k(0; 0) = J . In the next lemma, we nd a lower bound on the min-potential. Lemma 5.1 For all l 0 and for all j = 0; : : : ; (l) and for all t = 0; : : : ; T (l; j), P ( j;t l ) P ( 0) + ( ; )8<: l 1 X u=0 (u) X v=0 T (u; v) + j 1 X v=0T (l; v) + J Xi=1 (i) + t9=;+ l l 1 X u=0 (l) 1 X v=0 ln rv u j 1 X v=0 ln rv l J 1 Xi=0 ln ri + c( ; )k(l; j) + ( ; ) where c( ; ) and ( ; ) are as de ned in Step (1.0 ) and Step (1.4 ) of the algorithm, and ( ; ) is as de ned in Corollary 4.1. Proof. In view of Corollary 4.1 and (15), we have P ( j;t l ) P ( j;0 l ) + ( ; )t P ( j 1;T (l;j 1) l ) + ( ; )t ln rj 1 l + c( ; ) P ( j 1;0 l ) + ( ; ) (t+ T (l; j 1)) ln rj 1 l + c( ; ) 15 P ( 0;0 l ) + ( ; )0@t+ j 1 X v=0T (l; v)1A j 1 X v=0 ln rv l + c( ; )j P ( fl 1) + ( ; )0@t+ j 1 X v=0T (l; v)1A j 1 X v=0 ln rv l + ( ; ) + c( ; )j (by the translation of the objective cut) P ( 0;0 0 ) + ( ; )0@t+ j 1 X v=0T (l; v) + l 1 X u=0 (u) X v=0 T (u; v)1A+ ( ; )l j 1 X v=0 ln rv l l 1 X u=0 (l) 1 X v=0 ln rv u + c( ; )(k(l; j) J): Now in view of Lemma 4.2(9) (applied to the addition of the objective cut in the last step of phase I) and the de nition of ( ; ), we have P ( j;t l ) P ( J; (J)) + ( ; )0@t+ j 1 X v=0 T (l; v) + l 1 X u=0 (u) X v=0 T (u; v)1A+ ( ; )l j 1 X v=0 ln rv l l 1 X u=0 (l) 1 X v=0 ln rv u + c( ; )((k(l; j) J) + ( ; ) P ( 0) J 1 Xi=0 ln ri + c( ; )J + ( ; ) J Xi=1 (i) + ( ; )l + ( ; )0@t+ j 1 X v=0 T (l; v) + l 1 X u=0 (u) X v=0 T (u; v)1A j 1 X v=0 ln rv l l 1 X u=0 (l) 1 X v=0 ln rv u + c( ; )((k(l; j) J) + ( ; ) P ( 0) + ( ; )0@t+ J Xi=1 (i) + j 1 X v=0 T (l; v) + l 1 X u=0 (u) X v=0 T (u; v)1A+ ( ; )l J 1 Xi=0 ln ri j 1 X v=0 ln rv l l 1 X u=0 (l) 1 X v=0 ln rv u + c( ; )k(l; j) + ( ; ): 2 Now for xed u 0 and 0 v (u) 1 and for any y 2 ( v;T (u;v) u )int, we consider the slacks corresponding to the constraints of v;T (u;v) u at y (excluding the 2m constraints corresponding to the unit box [0; 1]m) represented as follows: srk(u; v; T (u; v); y) = crk(u; v; T (u; v)) (ark)Ty; 16 where 1 r (k) for 0 k u 1 and 1 r v for k = u, and ~ sj(u; v; T (u; v); y) = ~ cj(u; v; T (u; v)) (~ aj)Ty; for j = 1; : : : ; J . Since is contained in the cube [0; 1]m and the linear constraints are given by normal vectors of unit norm, we can easily conclude (see Go n et al. [14, Lemma 6.2]) that for k and r as de ned above, 0 srk(u; v; T (u; v); y) pm; (16) and for j = 1; : : : ; J , 0 ~ sj(u; v; T (u; v); y) pm: (17) We will now nd overestimators for rv u and rj for u 0, v = 0; : : : ; (u) 1 and for j = 0; : : : ; J 1 respectively that are easier to manipulate. We, therefore, construct the following matrices, using a construction due to Nesterov [20]. B0 = 8I; Bj+1 = Bj + ~ aj+1(~ aj+1)T m ; 0 j J 1; Bk(l;j+1) = Bk(l;j) + aj+1 l (aj+1 l )T m ; l 0; 0 j (l) 1: (Note that k(0; 0) = J and k(l + 1; 0) = k(l; (l)).) Lemma 5.2 The above matrices satisfy the following : Hj Bj; 0 j J 1; Hv u Bk(u;v); u 0; v = 0; : : : ; (u) 1: Proof. We give the proof for the case u 0 and v = 1; : : : ; (u) 1. The other cases can be proven very similarly. Hv u = u 1 X k=0 (k) X r=1 ark(ark)T [srk(u; v; T (u; v); yv u)]2 + v X r=1 aru(aru)T [sru(u; v; T (u; v); yv u)]2 + J Xj=1 ~ aj(~ aj)T [~ sj(u; v; T (u; v); yv u)]2 + (Y v u ) 2 + (I Y v u ) 2 17 + rf(yv u)rf(yv u)T [f(yv u) z(u)]2 + r2f(yv u) z(u) f(yv u) u 1 X k=0 (k) X r=1 ark(ark)T m + v X r=1 aru(aru)T m + J Xj=1 ~ aj(~ aj)T m + 8I (by (16), (17) and convexity of f) = Bk(u;v) Lemma 5.3 Let wk(u;v) = [(av+1 u )T [Bk(u;v)] 1av+1 u ]1=2 for u 0 and for v = 0; : : : ; (u) 1. Furthermore, let wj = [(~ aj+1)T (Bj) 1~aj+1]1=2 for j = 0; : : : ; J 1. Then wj rj ; j = 0; : : : ; J 1; wk(u;v) rv u; u 0; v = 0; : : : ; (u) 1: Proof. Again, we consider the case for u 0 and for v = 0; : : : ; (u) 1. The other case can be proven very similarly. (wk(u;v))2 = (av+1 u )T [Bk(u;v)] 1av+1 u (av+1 u )T [Hv u] 1av+1 u (by Lemma 5.2) = (rv u)2: Lemma 5.4 For any k 0, we have k X r=0(wr)2 2m2 ln 1 + k + 1 8m2 ! : Proof. Using a result of Nesterov [20], Go n et al. [14, Lemma 6.5] have achieved this upper bound. 2 18 We now consider stopping criterion 1 of the algorithm. This stopping rule implies that the total number of feasibility cuts produced by the algorithm does not exceed k ; , where k ; is the smallest index, k, satisfying 1 + 4m ln(1 + k 8m2 ) 1 + 2m+ k exp 2c( ; )k 1 + 2m+ k 2 ( ; ) 1 + 2m+ k! ( )2 2m : (18) Hence, the total number of feasibility cuts produced by the algorithm is at most O (m2=( )2), where the asterisk notation means that lower order terms are ignored. We will show that under Assumption 2, as soon as stopping criterion 1 is satis ed, the last feasible point found by the algorithm is an -solution of (CP ). We rst nd an upper bound on the min-potential of the sets of localization in the following lemma. Lemma 5.5 Let y be a solution of (CP ), > 0 be a given tolerance and 0 < < 1=2 be as in Assumption 2. Then z(l) f implies that P ( j;t l ) f1 + 2m+ k(l; j)g ln 2 for all l 0, j = 0; : : : ; (l) and t = 0; : : : ; T (l; j). Proof. Consider the ball B( y; ( =2)) =2. Then since z(l) f and 0 < < 1=2, we have z(l) f( y) f + f( y) =2 > =2 : Now, in view of the de nition of the min-potential, Assumption 2 and Corollary 2.2 we haveP ( j;t l ) = min y2 j;t l 8<: ln(z(l) f(y)) l 1 X u=0 (u) X v=1 ln(cvu(l; j; t) (avu)Ty) j X k=1 ln(ckl (l; j; t) (akl )Ty) J X k=1 ln(~ ck(l; j; t) (~ ak)Ty) m Xi=1 ln yi m Xi=1 ln(1 yi)) ln(z(l) f( y)) l 1 X u=0 (u) X v=1 ln(cvu(l; j; t) (avu)T y) 19 j X k=1 ln(ckl (l; j; t) (akl )T y) J X k=1 ln(~ ck(l; j; t) (~ ak)T y) m Xi=1 ln yi m Xi=1 ln(1 yi) ln 2 + (2m+ k(l; j)) ln 1 ( =2) (1 + 2m+ k(l; j)) ln 2 (since ( =2) =2): 2 Theorem 5.1 Under Assumption 2, as soon as stopping criterion 1 is satis ed, the last feasible test point found by the algorithm is an -solution of problem (CP ). Proof. Using Lemma 5.1, we can nd the following lower bound on the min-potential: P ( j;t l ) P ( 0) l 1 X u=0 (l) 1 X v=0 ln rv u j 1 X v=0 ln rv l J 1 Xi=0 ln ri + c( ; )k(l; j) + ( ; ); (19) where l 0, 0 j (l) and 0 t T (l; j). It now follows from Lemma 5.5 and (19) that as long as z(l) f , we have (1 + 2m+ k(l; j)) ln 2 2m ln 12 + l 1 X u=0 (u) 1 X v=0 ln rv u + j 1 X v=0 ln rv l + J 1 Xi=0 ln ri c( ; )k(l; j) ( ; ): Now by Lemma 5.3 and Lemma 5.4, we have (1 + 2m+ k(l; j)) ln 2 2m ln 1 2 + k(l;j) 1 X k=0 (wk)2 c( ; )k(l; j) ( ; ) 2m ln 1 2 + 2m2 ln 1 + k(l; j) 8m2 ! c( ; )k(l; j) ( ; ); which following the argument by Go n et al. [14, Theorem 6.1] we conclude that ( )2 4m 1 2 + 2m ln(1 + k(l;j) 8m2 ) 1 + 2m+ k(l; j) exp 2c( ; )k(l; j) 1 + 2m+ k(l; j) 2 ( ; ) 1 + 2m+ k(l; j)! : (20) 20 Thus, if inequality (20) is violated, we have z(l) f , which implies that f(yf l 1) f < since z(l) > f(yf l 1). This means that as soon as stopping criterion 1 is satis ed by the algorithm, the last feasible point found is an -solution. 2 We will now consider stopping criterion 2 of the algorithm. The following lemma will lead to an upper bound on the gap z(l) f , which is used in stopping criterion 2. Lemma 5.6 Let ya l be the exact analytic center of fl , where l 0 . Then z(l) f f1 + 2m+ k(l; (l))g(z(l) f(ya l )): Proof. Let n(l) = 2m+ k(l; (l)) and let fl be represented as follows: fl = fy : f(y) z(l); fk(y) 0; k = 1; : : : ; n(l)g: The exact analytic center ya l minimizes ( fl ; y). The necessary and su cient conditions for this minimum are 8><>: fk(ya l ) 0; k = 1; : : : ; n(l) Pn(l) k=1 xkrfk(ya l ) = rf(ya l ); x 0 xkfk(ya l ) = z(l) f(ya l ); k = 1; : : : ; n(l) : (21) Let fl = fy : fk(y) 0; k = 1; : : : ; n(l)g: Then by Wolfe's formulation of duality [24], the dual of minff(y) : y 2 fl g is given by max x;y 8<:f(y) + n(l) X k=1xkfk(y) : n(l) X k=1 xkrfk(y) = rf(y); x 09=; : Since (x; ya l ) is a feasible solution for this dual, we have f l f(ya l ) + n(l) X k=1 xkfk(ya l ) where f l is the optimal value of f on fl . But f f l since fl . Therefore, f(ya l ) f n(l) X k=1 xkfk(ya l ) = n(l)(z(l) f(ya l )) : 21 This implies that z(l) f z(l) f(ya l ) + f(ya l ) f (1 + n(l))(z(l) f(ya l )) Theorem 5.2 If stopping criterion 2 of the algorithm is satis ed at a feasible test point, then this point is an -solution of the problem. Proof. Let n(l) = 2m+ k(l; (l)). Then Lemma 5.6 together with Lemma 4.1(8) and the fact that in the algorithm 0 < (p2 1)2=2, imply that if yf l , where l 0, satis es stopping criterion 2, then z(l) f (1 + n(l))(z(l) f(ya l )) 1 + (2 p2)p ! (1 + n(l))(z(l) f(yf l )) 5 4(1 + n(l))(z(l) f(yf l )) : This means that f(yf l ) f < ; therefore, yf l is an -solution of the problem (CP ). 2 Now let ̂ = 4 (1 ) 5(1 + 2m+ k ; ) : (22) Then we have the following result: Lemma 5.7 If, for l 1, z(l 1) f(yf l 1) + 4 5(1 + 2m+ k(l 1; (l 1))) ; then for all j = 0; : : : ; (l) and for all t = 0; : : : ; T (l; j), we have P ( j;t l ) (1 + 2m+ k(l; j)) ln ̂ ; where = 2max(1; z(0) f ( z(0) f )) : 22 Proof. Note thatz(l) f z(l) f(yf l 1) = (1 )(z(l 1) f(yf l 1) (1 ) 4 5(1 + 2m+ k(l 1; (l 1))) ̂ : Now consider the ball B(ŷ; ( ̂=2)) ̂=2. Then z(l) f(ŷ) f + ̂ f(ŷ) ̂=2 : Also in view of Corollary 2.1, we have ( ̂=2) ̂ (z(0) f ) 2(z(0) f ) : Therefore, we have P ( j;t l ) = min y2 j;t l 8<: ln(z(l) f(y)) l 1 X u=0 (u) X v=1 ln(cvu(l; j; t) (avu)Ty) j X k=1 ln(ckl (l; j; t) (akl )Ty) J X k=1 ln(~ ck(l; j; t) (~ ak)Ty) m Xi=1 ln yi m Xi=1 ln(1 yi)) ln(z(l) f(ŷ)) l 1 X u=0 (u) X v=1 ln(cvu(l; j; t) (avu)T ŷ) j X k=1 ln(ckl (l; j; t) (akl )T ŷ) J X k=1 ln(~ ck(l; j; t) (~ ak)T ŷ) m Xi=1 ln ŷi m Xi=1 ln(1 ŷi) ln 2̂ + (2m+ k(l; j)) ln 1 ( ̂=2) ln 2̂ + (2m+ k(l; j)) ln 2(z(0) f ) ̂ ( z(0) f ) (1 + 2m+ k(l; j)) ln ̂ : 2 23 We can now derive an upper bound on the total number of iterations performed by the algorithm. Theorem 5.3 The total number of iterations in phase I and phase II of the algorithm to nd an -solution is O (m2 + k ; ) ln (m+ k ; ) (1 ) ! where is as given in Lemma 5.7 and k ; is the smallest index satisfying (18) and the constant factor in O( ) depends on and on the choice of only. Proof. As long as, z(l 1) f(yf l 1) + 4 5(1 + 2m+ k(l 1; (l 1))) ; where l 1, by Lemma 5.1 and Lemma 5.7, we have (1 + 2m+ k(l; j)) ln ̂ 2m ln 1 2 l 1 X u=0 (l) 1 X v=0 ln rv u j 1 X v=0 ln rv l J 1 Xi=0 ln ri + ( ; )8<: l 1 X u=0 (u) X v=0 T (u; v) + j 1 X v=0 T (l; v) + J Xi=1 (i) + t9=; + ( ; )l + c( ; )k(l; j) + ( ; ): Hence, in view of Lemma 5.4, we have ( ; )8<: l 1 X u=0 (u) X v=0 T (u; v) + j 1 X v=0T (l; v) + J Xi=0 (i) + t9=;+ ( ; )l (1 + 2m+ k(l; j)) ln ̂ + 2m ln 12 c( ; )(k(l; j) ( ; ) + l 1 X u=0 (l) 1 X v=0 ln rv u + j 1 X v=0 ln rv l + J 1 Xi=0 ln ri (1 + 2m+ k(l; j)) ln ̂ + 2m ln 12 c( ; )k(l; j) ( ; ) + 1 2 k(l;j) 1 X r=0 ln(wr)2 (1 + 2m+ k(l; j)) ln ̂ + 2m ln 12 c( ; )k(l; j) ( ; ) +m2 ln 1 + k(l; j) 8m2 ! : 24 Now since 0 < < 1 and the total number of feasibility cuts is bounded by k ; , we have l 1 X u=0 (u) X v=0 T (u; v) + j 1 X v=0 T (l; v) + J Xi=0 (i) + t+ l 1 ( ; ) "(1 + 2m + k(l; j)) ln ̂ + 2m ln 1 2 c( ; )k(l; j) ( ; ) + m2 ln 1 + k(l; j) 8m2 !# 1 ( ; ) "(1 + 2m + k ; ) ln ̂ + 2m ln 12 + jc( ; )jk ; ( ; ) + m2 ln 1 + k ; 8m2!# : (23) Note that the left hand side of inequality (23) represents the total number of translations performed during the algorithm. Let T ; be the smallest index that is greater than the right hand side of inequality (23). Suppose that stopping criterion 1 is not satis ed. As soon as inequality (23) is violated, we have z(l 1) f(yf l 1) 4 5(1 + 2m+ k(l 1; (l 1))) ; i.e., stopping criterion 2 is satis ed. Hence the total number of translations (both objective and constraint) does not exceed T ; . Since the total number of feasibility cuts is k ; , in no more than T ; +k ; iterations, stopping criterion 2 will be satis ed. If stopping criterion 1 is satis ed, then inequality (23) holds and again the total number of iterations will be bounded by T ; +k ; . Since this upper bound is O( (m2+k ; ) ln ̂ ), the result follows. 2 We now consider phase I of the algorithm. Theorem 5.4 The total number of iterations in phase I of the algorithm to nd a feasible point w0 2 is O((m2+ k ) ln k ) where k is the smallest index, j satisfying 2 m 12 + 2m ln(1 + j 8m2 ) 2m+ j e 2c( ) j j+2m with c( ) = 2 p2 + lnp2 + ln[((p2 1)2 )] < 0; and the constant factor in O( ) depends on the choice of only. 25 Proof. We note that in phase I, = 1. Now under Assumption 1, and by a very similar argument to that in Lemma 5.1, we can conclude that P ( j;t) P ( 0) + (1; ) j Xi=1 (i) j 1 Xi=0 ln ri + c( )j P ( 0) j 1 Xi=0 ln ri + c( )j; where ( ; ), with = 1, is as de ned in Corollary 4.1. Furthermore, by Lemma 6.1 of Go n et al. [14], we have P ( j;t) (2m+ j) ln 1 for all j = 1; : : : ; J and for all t = 0; : : : ; (j). Therefore, by a very similar discussion to that of Go n et al. [14, Theorem 6.6], we can conclude that the total number of feasibility cuts in phase I, J , does not exceed k (an O (m2= 2) number). Furthermore, by Lemma 5.4, j Xi=1 (i) 1 (1; ) 24(2m+ j) ln 1 + 1 2 j 1 Xi=0 ln(wi)2 c( )j35 5 4 "(2m+ k ) ln 1 +m2 ln 1 + k 8m2! c( )k # : Since, violation of this inequality contradicts Assumption 1, the total number of translations does not exceed where is the smallest index that is greater than the right hand side of the above inequality. The overall complexity of this phase now follows. 6 Update to a New Center In each iteration of phase I or phase II of the algorithm, we need to calculate a -approximate analytic center (with 0 < (p2 1)2=2) of either an outer approximation of or a set of localization. In this section, we will study how to calculate a new approximate analytic center in the following four cases: 1. When a feasibility cut is added; 26 2. when a feasibility cut shifted; 3. when the nonlinear objective cut is added; and 4. when the objective cut is shifted. We will also give upper bounds on the total number of (damped) Newton steps required to recenter. Throughout this section, we assume that N is de ned by (6). Theorem 6.1 In the algorithm, the total number of (damped) Newton steps to update to a new -approximate analytic center after addition of a linear cut, is at most O(1), with O(1) depending on and only. Proof. We choose y+ according to (11) of Lemma 4.3 and use the same notation as in that lemma. Hence, we have ( +; y+) P ( +) c( ; ) c( ; ) (24) where c( ; ) and c( ; ) are given by (13) and (14). Now let q( +; y) be the Newton direction of ( +; y) at y. By Lemma 2.3, as long as p kg( +; y)k H( +;y) there is the following minimal decrease in the min-potential at each iteration. ( +; y) ( +; y + q( +; y)) !( ) (25) where !( ) = f ln(1 + )g= (26) with given by (4). It now follows from (24) and (25) that, starting from y+, in no more than & c( ; ) c( ; ) !( ) ' damped Newton steps, a -approximate analytic center of + is calculated. Furthermore, by Lemma 2.4, at most another N (depending on only) full Newton steps are needed to calculate a -approximate analytic center of +. 2 27 To study how to update to a new center after the translation of a linear cut, we rst recall the following result from Nemirovskii [19]. This result is stated there for 1-selfconcordant functions, yet can be easily extended to -self-concordant functions. A very similar result can also be found in [21, Proposition 2.2.3 ]. Lemma 6.1 Let be de ned by (1). Suppose that ( ; y) is -self-concordant. Then, for all h such that y + h 2 int, ( ; y + h) ( ; y) + hg( ; y); hi + 1 ( p khkH( ;y)); (27) where (s) = ln(1 s) s. Theorem 6.2 In the algorithm, the total number of (damped) Newton iterations to update to a new -approximate analytic center after translation of a linear cut is at most O(1), where O(1) depends on and on the choice of only. Proof. Let be de ned by (1) with yc a -approximate analytic center of it. Suppose that some of the constraints in are linear and that ( ; y) is -self-concordant. Consider the case in which a linear constraint of (without loss of generality the n-th constraint given by hn(y) = aTny cn) is translated all the way to yc, i.e., is updated to = fy 2 Rm : hj(y) 0; j = 1; : : : ; n 1; aTny aTnycg: Note that ( ; y) ln(cn aTny) = ( ; y) ln(aTn (yc y)): Let y be chosen as y = yc [H( ; yc)] 1an 3p kank H( ;yc) : By Lemma 4.3, we have y 2 int and ( ; y) ln(cn aTn y) ( ; yc) ln kank H( ;yc) + c( ; ); or ( ; y) ( ; yc) + ln(cn aTn y) ln kank H( ;yc) + c( ; ): (28) 28 On the other hand, by Lemma 6.1, for all h 2 Rm such that yc + h 2 int, and using the same de nition for  as in that lemma, we have ( ; yc + h) ( ; yc) + hg( ; yc); hi+ 1 ( p khkH( ;yc)): Let r = p khkH( ;yc). Then ( ; yc + h) ( ; yc) kg( ; yc)k H( ;yc)khkH( ;yc) + [ ln(1 + r) + r]= ( ; yc) r= + [ ln(1 + r) + r]= : (29) Now let h be such that yc + h 2 int. Note that in this case, we have aTnh 0. This together with (29) imply that for all h such that yc + h 2 int, we have ( ; yc + h) = ( ; yc + h) + ln(cn aTn (yc + h)) ln( aTnh) ( ; yc) r= + [ ln(1 + r) + r]= + ln(cn aTnyc) ln kank H( ;yc) ln(r=p ): Therefore, P ( ) ( ; yc) ln kank H( ;yc) + ln(cn aTnyc) + ~ c( ; ); (30) where ~ c( ; ) = min r>0 f[ ln(1 + r) + (1 )r]= ln(r=p )g: It is easy to see that ~ c( ; ) is well de ned. Now (28) and (30) imply that ( ; y) P ( ) c( ; ) ~ c( ; ) + ln cn aTn y cn aTnyc! : (31) But k y yckH( ;yc) = 1 3p : Also, by the de nition of H( ; yc), we have anaTn (cn aTnyc)2 H( ; yc): Therefore, aTn (yc y) cn aTnyc 1 3p which implies that cn aTn y cn aTnyc 1 + 1 3p : (32) 29 Now (31) and (32) imply that ( ; y) P ( ) ln 1 + 1 3p !+ c( ; ) ~ c( ; ): (33) Hence, starting at y, it takes at most 2666 ln(1 + 1 3p ) + c( ; ) ~ c( ; ) !( ) 3777 damped Newton iterations to nd a -approximate analytic center of . Here, !( ) is as in (26). Another N full Newton steps is required to calculate a -approximate analytic center of . 2 At the end of phase I, the cut corresponding to the objective, which is a nonlinear cut is added. The next theorem shows how to update to a new center in this case. Theorem 6.3 Consider w0 2 int which is a -approximate analytic center of J; (J). It takes O(1) (damped) Newton steps to calculate a -approximate analytic center of 0;0 0 . Here, O(1) depends on and only. Proof. By Lemma 4.2(10), we have kg( 0;0 0 ; w0)k H( 0;0 0 ;w0) kg( J; (J); w0)k H( J; (J) ;w0) + p : Recall from the algorithm that 0 < (p2 1)2=2. Hence kg( 0;0 0 ; w0)k H( 0;0 0 ;w0) 2 p < 1 3p : It now follows from Lemma 2.2 that ( 0;0 0 ; w0) P ( 0;0 0 ) d( ; ) where d( ; ) = 4 2 f1 (9 2 )2g : (34) Now using the damped Newton Method, starting at w0, it takes no more than &d;( ; ) !( ) ' 30 damped Newton steps (with !( ) de ned by (26)) to nd a -approximate analytic center of 0;0 0 . Finally, by Lemma 2.4, it takes another N Newton steps to calculate a -approximate analytic center of 0;0 0 . 2 Finally, we consider the case where the objective constraint is shifted. The next lemma is due to Den Hertog [5, Theorem 3.4] which is also a generalization of an inequality of Vaidya [23] for the LP-case to the convex case. Lemma 6.2 Consider the bounded set ( ) = fy 2 Rm : gi(y) di; i = 1; : : : ; n 1; gn(y) g (35) with a nonempty interior, where gi : Rm ! R; i = 1; : : : ; n are convex and di erentiable. Then as decreases, f gn(ya )g decreases monotonically. Here ya is the exact analytic center of ( ). Theorem 6.4 Consider the bounded set = fy 2 Rm : gi(y) di; i = 1; : : : ; ng with int 6= ;, where the functions gi : Rm ! R; i = 1; : : : ; n are convex and twice continuously di erentiable. Let = fy 2 Rm : gi(y) di; i = 1; : : : ; n 1; gn(y) dn( )g where dn( ) = dn (dn gn(yc)), 0 < < 1 and yc is a -approximate analytic center of . Furthermore, suppose the potential functions associated with and are -self-concordant. Then the total number of (damped) Newton iterations to nd a -approximate analytic center of is at most O(1), where O(1) depends on , and only. Proof. By Lemma 2.2, we have ( ; yc) P ( ) ( ; ) (36) where ( ; ) = 2 f1 (9 4 )2g : (37) 31 Let ( ) be as de ned by (35) and let ya and ya be the exact analytic centers of and ( ) respectively. Consider ~ (y; ) = ( ( ); y) P ( ( )): We note that ~ (y; dn) = ( ; y) P ( ) ~ (y; dn( )) = ( ; y) P ( ): We will use an argument due to Den Hertog [5, Theorem 3.6] to nd an upper bound on ~ (y; dn( )). By Mean Value Theorem, there exists ̂ 2 (dn( ); dn) such that for all y 2 , ~ (y; dn( )) = ~ (y; dn) d d ~ (y; ) =̂ (dn dn( )): But d d ~ (y; ) = 1 gn(y) d d P ( ( )) and d d P ( ( )) = n 1 Xi=1 rgi(ya )dya d di gi(ya ) + rgn(ya )dya d 1 gn(ya ) = 1 gn(ya ) by (21). Therefore, d d ~ (y; ) = 1 gn(y) + 1 gn(ya ) which in view of Lemma 6.2 implies that ~ (y; dn( )) = ~ (y; dn) ( 1 gn(y) + 1 gn(ya )) =̂ (dn dn( )) ~ (y; dn) ( 1 dn( ) gn(y) + 1 dn gn(ya)) (dn dn( )) Now by (36) and the de nition of dn( ), we have ~ (yc; dn( )) ( ; ) ( 1 + dn gn(yc) dn gn(ya)) ( ; ) + 8<: 1 1 1 1 + (2 p2)p 9=; by (Lemma 4.1): 32 Thus,( ; yc) P ( ) ( ; ) +8<: 1111 + (2 p2)p 9=; :(38)Hence, in no more than26666 ( ; ) + n 1111+ =((2 p2)p )o!( )37777damped Newton iterations (!( ) as in (26)), a -approximate analytic center ofis calculated. Since < , we can use full Newton steps according to Lemma 2.4to calculate a -approximate analytic center of in at most another N full Newtonsteps.Corollary 6.1 In the algorithm, the total number of (damped) Newton steps to up-date to a new -approximate analytic center after a translation of the objective con-straint, is O(1), with O(1) depending on , and only.We can now conclude the following result:Theorem 6.5 The column generation algorithm nds an -solution of the problem(CP ) in at mostO (m2 + k ; ) ln (m+ k ; )! ;(damped) Newton iterations, where is as given in Lemma 5.7 and k ; is the smallestindex satisfying (18) and the constant factor in O( ) depends on and on the choiceof and only.Remark. If the oracle can generate a nite number of cutting planes, e.g., boundedabove by N , with N m, then the overall complexity is O(N ln(N = )), i.e. thealgorithm is polynomial in N . In this case stopping criterion 1 can be dropped.On the other hand, if the oracle can generate an in nite number of cutting planes,then the overall complexity is O ( m2( )2 ln(m23 2 )), i.e., the algorithm is a polynomialapproximation scheme in the dimension of the space,m. A polynomial approximationscheme means that for every > 0, the method is polynomial in the dimension of thespace.33 References[1] A. Altman and K.C. Kiwiel, A note on some analytic center cutting plane meth-ods for convex feasibility and minimization problems, Computational Optimiza-tion and Applications, 5 (1996), pp. 175-180.[2] D.S. Atkinson and P.M. Vaidya, A cutting plane algorithm for convex program-ming that uses analytic centers, Mathematical Programming, 69 (1995), pp. 1-43.[3] O. Bahn, O. Du Merle, J.L. Go n and J.P. Vial, A cutting plane method fromanalytic centers for stochastic programming, Mathematical Programming, 69(1995), pp. 45-73.[4] O. Bahn, J.L. Go n, J.P. Vial and O. Du Merle, Experimental behavior of aninterior point cutting plane algorithm for convex programming: an application togeometric programming, Discrete Applied Mathematics, 49 (1994), pp. 3-23.[5] D. Den Hertog, Interior-Point Approach to Linear, Quadratic and ConvexProgramming, Algorithms and Complexity, Kluwer Publishers, Dordrecht, TheNetherlands, 1994.[6] D. Den Hertog, J. Kaliski, C. Roos and T. Terlaky, A logarithmic barrier cuttingplane method for convex programming, Annals of Operations Research, 58 (1995),69-98.[7] J. Kaliski, D. Haglin, C. Roos and T. Terlaky, Logarithmic barrier decompositionmethods for semi-in nite programming, Research Report, No. 96-51, Faculty ofTechnical Mathematics and Informatics, Delft University of Technology, TheNetherlands, 1996.[8] H. G. Eggleston, Convexity, Cambridge University Press,1969.[9] J.L. Go n, Convergence rates of the ellipsoid method on general convex func-tions, Mathematics of Operations Research, vol. 8. No. 1. (1983), pp. 135-150.[10] J.L. Go n, J. Gondzio, R. Sarkissian and J.P. Vial, Solving nonlinear multi-commodity problems by the analytic center cutting plane method, MathematicalProgramming (to appear ).34 [11] J.L. Go n, A. Haurie and J.P. Vial, Decomposition and nondi erentiable op-timization with the projective algorithm, Management Science 37, (1992), pp.284-302.[12] J.L. Go n, A. Haurie, J.P. Vial and D.L. Zhu, Using central prices in the de-composition of linear programs, European Journal of Operations Research, 64(1993), pp. 393-409.[13] J.L. Go n, Z.Q. Luo and Y. Ye, On the complexity of a column generation algo-rithm for convex or quasiconvex problems, in Large Scale Optimization: State ofthe Art, W.W. Hager, D.W. Hearn and P.M. Pardalos, editors, Kluwer AcademicPublishers, 1993, pp. 187-196.[14] J.L. Go n, Z.Q. Luo and Y. Ye, Complexity analysis of an interior point cuttingplane method for convex feasibility problems, SIAM Journal on Optimization, vol6, No. 3 (1996), pp. 638-652.[15] J.L. Go n and J.P. Vial, Shallow, deep and very deep cuts in the analytic centercutting plane method, Logilab Technical Report, University of Geneva, Switzer-land, 1996.[16] K.C. Kiwiel, E ciency of the analytic center cutting plane method for convexminimization, SIAM Journal on Optimization (to appear).[17] Z.Q. Luo and J. Sun, An analytic center based column generation algorithmfor convex quadratic feasibility problems, Preprint, Department of Electrical andComputer Engineering, McMaster University, Hamilton, Ontario, Canada, 1995.[18] Z.Q. Luo and J. Sun, Cutting surfaces and analytic center: A polynomial al-gorithm for a convex feasibility problem de ned by self-concordant inequalities,Preprint, Department of Electrical and Computer Engineering, McMaster Uni-versity, Hamilton, Ontario, Canada, 1996.[19] A.S. Nemirovskii, Interior-Point Polynomial Methods in Convex Programming,Lecture notes, Faculty of Industrial Engineering and Management, Israel Insti-tute of Technology, Israel, 1994.[20] Y.E. Nesterov, Complexity estimates of some cutting plane methods based on theanalytical barrier, Mathematical Programming, vol. 69 (1995), pp. 149-176.35 [21] Y.E. Nesterov and A.S. Nemirovskii, Interior-Point Polynomial Algorithms inConvex Programming, SIAM, Philadelphia, 1994.[22] G. Sonnevend, New algorithms in convex programming based on a notion of \cen-ter" (for systems of analytic inequalities) and on rational extrapolation, in Trendsin Mathematical Optimization: Proceedings of the 4th French-German confer-ece on Optimization in Irsee, West Germany, April 1986, K.H. Ho man, J.B.Hiriart-Urruty, C. Lemarechal and J. Zowe (eds.), International Series of Numer-ical Mathematics, 84, Birkhauser-Verlag: Basel, Switzerland, 1988, pp. 311-327.[23] P.M. Vaidya, An algorithm for linear programming which requires O(((m+n)n2+(m+n)1:5n)L) arithmetic operations, Mathematical Programming, 47 (1990), pp.175-201.[24] Ph. Wolfe, A duality theorem for nonlinear programming, Quarterly of AppliedMathematics, 19 (1961), pp. 239-244.[25] Y. Ye, A potential reduction algorithm allowing column generation, SIAM Journalon Optimization, 2 (1992), pp. 7-20.[26] Y. Ye, Complexity analysis of the analytic center cutting plane method that usesmultiple cuts, Technical Report, Department of Management Sciences, Universityof Iowa, Iowa City, IA, 1994.36 7 AppendixIn this appendix, we give a proof of Lemma 4.3 which is an extension of Lemma 2.1of Nesterov [20] given for strongly 1-self-concordant functions. The proof is verysimilar, yet some constants change due to the fact that our ( ; y) is assumed to be-self-concordant. We need to state two lemmas that will be used in the proof. Thefollowing lemma is taken from Nesterov and Nemirovskii [21, Theorem 2.1.1].Lemma 7.1 Let be de ned by (1), ( ; y) be (strongly) -self-concordant on int,and x 2 int. Then for every y 2 such thatr = p kx ykH( ;x) < 1(39)we have(1 r)2H( ; x) H( ; y)1(1 r)2H( ; x):Furthermore, y 2 int.The next lemma is a trivial extension of Theorem 2.3 of [20]. The result there isfor strongly 1-self-concordant functions. We can simply extend it to strongly -self-concordant functions by noticing that if ( ; y) is -self-concordant, then ( ; y) is1-self-concordant.Lemma 7.2 Under the same assumptions as in Lemma 7.1, we have the following:1. ( ; y)( ; x) + h g( ; x); y xi (ln(1 r) + r):2. ( ; x)( ; y) + h g( ; y); x yi+ ln(1 r) + r=(1 r):3. p kg( ; y) g( ; x) H( ; x)(y x)k H( ;x) r(1=(1 r) 1):Proof of Lemma 4.3:Consider a central pathy(t) = arg minf ( ; u) : ha+; yc ui = tg; 0 t t ;wheret = supfha+; yc ui : u 2 g:37 As is argued in [20], this path is well de ned. By the rst order optimality conditions,for any t 2 [0; t ), there exists (t) 0 such thatg( ; y(t)) + (t)a+ = 0:(40)Let ya+ be the exact analytic center of +. Then for any t 2 [0; t ), we haveP ( +)( ; y(t)) + hg( ; y(t)); ya+ y(t)i ln= ( ; y(t)) + hg( ; y(t)); yc y(t)i+ (t) ln( )with = ha+; yc ya+i > 0. Using the inequalityln 1 + ln ;valid for all positive and , we getP ( +) ( ; y(t)) + hg( ; y(t)); yc y(t)i+ 1 + ln (t):(41)Since yc is a -approximate analytic center of , it follows from Lemma 2.1 thatp ky(0)yckH( ;yc) 1 1p2 r:We can choose t in such a way that this inequality becomes an equality:p ky(0)yckH( ;yc) = r:We now nd a lower bound on (t). By Lemma 7.2(3) and de nition of r, we have(t)t = h g( ; y(t)); y(t) yci= h g( ; y(t)) g( ; yc) H( ; yc)(y(t) yc); y(t) yci (42)+ hg( ; yc); y(t) yci+ r2rp kg( ; y(t)) g( ; yc) H( ; yc)(y x)k H( ;yc)p kg( ; yc)kH( ;yc)r + r2r2 11 r 1 r + r2= r((p2 1)2 );(43)38 (t)(ka+k H( ;yc))2 = (t)(ka+kH( ;yc))2= hH 1( ; yc)a+; g( ; y(t))it hH 1( ; yc)a+; g( ; y(t))i+hH 1( ; yc)a+; g( ; yc) +H( ; yc)(y(t) yc) g( ; yc)it+ r 11 r 1 ka+k H( ;yc):(44)Substituting inequality (44) into (43) by eliminating t, we get(t)ka+k H( ;yc) (t)ka+k H( ;yc) + + r 11 r 1r((p2 1)2 ):This quadratic inequality gives the following lower bound for (t).(t) (p2 1)2p ka+kH( ;yc) :(45)(One can easily check that the right-hand side of (45) satis es the quadratic inequalityas an equality.) Now by (41), Lemma 7.2(2) and (45), we havep( +)( ; yc) 1ln(1 r) + r1 r + 1 + ln (t)( ; yc) ln ka+kH( ;yc) + c( ; )wherec( ; ) = 1(p2 lnp2 1)= + ln[((p2 1)2 )=p ]:To get the upper bound for P ( +), we use Lemma 7.2(1) as follows.( +; y+) = ( ; y+) lnha+; yc y+i= ( ; y+) ln ka+kH( ;yc) + ln(3p )( ; yc) + hg( ; yc); y+ yci (ln(2=3) + 1=3)=ln ka+kH( ;yc) + ln(3p )( ; yc) + =(3 ) (ln(2=3) + 1=3)= lnka+kH( ;yc) + ln(3p )= ( ; yc) ln ka+kH( ;yc) + c( ; )wherec( ; ) = =(3 ) + ln(3p ) (ln(2=3) + 1=3)= :239